DOCUMENT RESUME 



ED 297 0?1 



TM Oil 989 



AUTHOR 
TITLE 

SPONS A -ENCY 

PUB DATE 
GRANT 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Kieren, Dianne K.; Munro, Brenda 

The Observational Recording Dilemma. 

Social Sciences and Humanities Research Council o£ 

Canada, Ottawa (Ontario). 

[85] 

410-84-0256 

26p.; For related documents, see TM Oil 987-988. 
Reports - Research/Technical (143) 

MF01/PC02 Plus Postage. 

Comparative Analysis; *Decision Making; *Family Life; 
Family Problems; *Interrater Reliability; 
*Observation; Problem Solving; *Research Methodology; 
Videotape Recordings 

Coding; *Observation Techniques; *Transcription 
Method 



ABSTRACT 

Decision making about an observational recording 
system for family interaction research is crucial. Alternative 
coding-recording methods and combinations thereof are discussed, 
including: (1) paper-and-pencil on-site method; (2) video-tapes; (3) 
paper-and-pencil and mechanical coding devices; (4) transcripts; and 
(5) transcripts combined with video-tapes. Data from a family problem 
solving study illustrate the type of questions that must be answered 
to arrive at a decision for a research study. Time requirements, 
number of units coded, interrater reliabilities for transcript 
methods, and data profile comparisons are reported. The on-site 
method was the least costly^ but revealed the most discreprait data 
profile compared with the other methods. Transcript methods were most 
costly, but also provided higher levels of precision in detailing 
family interactions. Video alone appeared to offer the best choice 
for coding-recording considering cost, comprehensiveness, and 
accuracy. Further testing of such methods is suggested to assist 
researchers in making decisions about coding-recording methods. Five 
tables are included. (TJH) 
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ABSTRACT 

THE OBSERVATIONAL RECORDING DILEMMA 



Decision making about an observational recording system for 
family interaction research is crucial. Issues of cost and 
quality of data are primary. The present paper discusses four 
alternative coding-recording methods: on-site ; paper and 
pencil; video only ; paper and pencil and mechanical coding 
devices; t ranscript only : paper and pencil; transcript plus 
vlde£: individual coder, paper and pencil and consensus. Data 
from a family problem solving study ar<i used to Illustrate the 
type of questions which must be answered to arrive at a 
decision for a research study. Time requirements, 

number of units coded, Interrater reliabilities for transcript 
methods, and data profile comparisons are reported. The 
on-site method was the least costly but revealed the most 
discrepant data profile compared with the other methods. 
Transcript methods were most costly but also provided higher 
levels of precision in detailing family Interaction. Video 
alone appeared to offer the best choice for coding-recording 
considering both cost, comprehensiveness and accuracy. Further 
testing of such methods is suggested to assist researchers in 
making decisions about coding-recording methods. 
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THE OBSERVATIOWAL RECORDING DILEMMA 



Introduction 

Recently, the amount of observational study of family and 
marital functioning lias Increased (See Fllslnger, 1983; 
Steinglass, 1979; Stelnglass & Tlslenko, 1983). The growing 
desire to gain the most accurate picture of family interaction 
combined with the decline in available grant funds necessitates 
careful selection of research design, methodology and type of 
analysis. Researchers must make many decisions in order that 
the ensuing study might be the most representative possible 
within economic constraints. Thus, doing observational 
research Involves the resolution of many methodological issues* 

One issue involves deciding upon the most appropriate, 
accurate and manageable recording system given the nature of 
the interaction being measured, the type and complexity of the 
code and the funding resources available. A number of choices 
are available: on-site observation and coding using either a 
paper and pencil or a mechanical recording device; observation 
and coding from videotape with or without transcription; coding 
from audiotape with or without transcription; or consensus 
coding using video, audio and/or transcript. Each method has 
its advantages and limitations. The use of transcripts 
increases the time and other costs. This method, however, 
provides greater precision in the coding process. 

The newer mechanical coding instruments (e.g. 0S3 and Data 
Myte) may be used on-site and/or with video and have the 
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potential of eliminating the transcribing step. 

These nethods require different training processes and 
challenge the researcher in assessing coding reliability 
especially when the code is a complex one. The question posed 
in this paper is: ''Do these less costly coding-recording 
methods obtain the same picture of family interaction as more 
costly ones? " 

Frequently the evaluation of different methods has been 
limited to a focus on intercoder consistency or reliability. 
Hartmann and Gardner (1981) defined the term observer 
reliability in terms of two related yet distinct paradigms, 
observer accuracy and Interobserver agreement, reliability or 
consistency* The accuracy paradigm compares what is assumed to 
be c flawed source of data with what is assumed to be an 
accurate or error less evaluation of an occurrence. 
Interobserver agreement compares two data sources that are 
af.sumed to be flawed with each other. In both cases, one code 
of a particular behavioral unit is compared with another code 
of the same behavioral unit. However, this type of comparison 
is not always possible when observing ongoing family 
intfecaction without the help of a transcript. When coders must 
decide upon the coding unit as well as the type of behavior, 
the number of interactions coded may vary and Interrater 
reliabilities may be flawed. 

Within the present investigation data profile reliability 
will be used in addition to assessing' coder accuracy and 
agreement, in the comparison of recording methods. That is, 

Er|c 7 
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while the number of coded Interactions might differ among 
recording techniques, the question posed, li whether the 
picture or profile of the family's interaction is similar. If 
there is data profile reliability, the proportion of codes in 
different categories should be the same or similar regardless 
of the method used* For example, a researcher using a 
transcript of an observational session is usually asked to code 
already identified coding units while an on site coder must 
identify coding units from the flow of interaction. Thus the 
coder might identify only a sampling of the coding units 
identified on a transcript. In addition, the on-site coder has 
no chance for a re-rtview. Even so, the portrait oif profile 
obtained of family interaction utilizing these two methods 
could be similar and would provide e compelling rationale for 
the choice of this r?-cording method over others. 

Thus, the purpose of thi& investigation was to conduct a 
limited test of these recording methods in an ongoing study of 
family problem solving interaction to provide a better 
rationale for the selection and use of a particular recording 
method in subsequent studies. Fully trained coders were used 
to determine the relative accuracy and time requirements of 
various recording methods. Comparisons of the methods were 
made based on the following questions: 

1) How do the number of ratings vary using different 
coding-recording methods? 

2) How does the time investaent for each method compare? 
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3) How does the Interrater reliability of the different 
recording methods uclng transcripts compare to a 
consensus code? 

4) How do the family profiles of family problem solving 
Interaction which emerge from these methods differ? 

A Review of Recording Methods 

Perhaps the most frequently used method of recording Is the 
on-site paper and pencil technique. Coders using only paper 
and pencil are faced with the formidable task of being able to 
Identify ar.d code behaviors in many response categories in a 
split second of time. Because the difficulty of this task has 
to be noted, commercial companies have come to the aid of 
researchers (Holm, 19Si) by producing machines which 
theoretically allow the coder to more rapidly and accurately, 
record observed behavior. Perhaps the most frequently used 
machines are the Datamyte, the 053 and the SSR System 7 
(Fllslnger, 1983). 

Another observational aid is the video recorder. 
Utilization of video recorded family interaction, allows coders 
to utilize many recording techniques. For example, coders 
eight code using paper and pencil, machines, or transcripts of 
the session. The distinct advantage of vidso is being able to 
turn the tape back in order to verify findings. 

While all of these recording techniques are viable, each 
has its distinct strengths and weaknesses which must be weighed 
by individual researchers dependent on their particular needs. 
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On one hand, researchers want to gather information that Is as 
reliable as la humanly possible. This means finding a 
technique where an optimal amount of family interaction 
information can be gleaned from observational settings with a 
methodological provision for testing the reliability of 
obtained information . 

Hartmann and Gardner (1981) discuss the importance of 
reliability in research in order to maintain as they state, the 
-quality of our research" (p. 184). It is Important that 
measurement be 1) accurate - correspond to the observed events, 
2) precise - that random error noise be reduced, and 3) 
generalizable - that obtained observations are representative 
of the real world and thus could be duplicated by other 
researchers . 

However, realistically, the economy of research projects is 
also an issue. Observational research is time consuming, both 
prior to and subsequent to the family interaction sessions. In 
most cases time and money constraints are a reality and 
researchers must do the best research they can within financial 
and titae boundaries. Thus research needs to be done without 
sacrificing the accuracy,, precision and generalizabili ty of the 
information obtained . 
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Methodology 

Coding Methods 

The present paper includes a report of the analysis of two 
problem situations comprising approximately 200 behavioral 
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codes In each situation from a larger study of family problem 
solving Interaction of three person family groups (Kleren & 
Hurlbut, 1985). In this analysis four different methods of 
coding and recording family problem solving interaction were 
compared in order to answer the research questions posed. The 
methods Included: 

1) coding on-site: one coder using a data logging 
instrument and one using a paper and pencil technique; 

2) coding using video; one coder using the data logging 
instrument and the other using paper and pencil; this was 
done one week after the initial on-site coding session; 

3) transcript only: paper and pencil; 

4) coding using transcript plus video; both coders using 
paper and pencil recording methods; and both coders using 
paper and pencil techniques and dlFCussing until 
consensus was reached. 

Coders were randomly assigned to the different methods In 
procedures one and two. 

Problem Solving Interaction 

The problem solving task for the family involved a revealed 
difference task developed by Kleren, Hurlbut, Gora, & Lehman 
(1985). Each family member was asked to independently answer a 
series of questions about nine common family problem solving 
situations. The questions were: What is the problem? Has a 
situation like this one ever happened in your family? Who is 
most responsible for the proMem? What should be done? Who 
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r^hould make the final decision? Once each pcroon in the family 
had independently answered the questionnaire their responses 
were reviewed by the researchers. Three of these situations 
were selected based upon salience for the family and some level 
of revealed difference between tnenber's responses. The family 
was allowed up to ten minutes to discuss each situation and 
come to a family consensus on answers to the questions. The 
interaction was videotaped. Thz present comparison was based 
upon the analysis of two of these situations since the multiple 
coding of these situations included the coding of over 1600 
behavioral units. 

General Coding Procedures 

A twenty six category code was developed which specifically 
tapped family problem solving behavior (Kieren, 1985). Coders 
were required to code each complete thought unit using a four 
digit code. The code represented 1) the sender, 2) the 
receiver and 3-4) the specific two digit code for each problem 
solving behavior . 

Two trained coders were utilized to code the data. 
Training involved an extensive process of an average of 55 
hours. This involved studying the code, achieving suff ciently 
high content accuracy (90Z), practicing using transcripts as 
uell as tri^.tiscripts plus video, learning to use a data logging 
inctr ^ (0S3)-, practicing using a data logging Instrument 
and • ^ith trainers. Interrafcer reliability was 

dett < uring training using a kappa statistic (Cohen, 
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1960) applied to a trial when coders used video plus 
transcripts since such a method requires comparison of similar 
coding units. The reliability achieved during training was .75 
which compares favorably with levels reported by other 
interaction researchers using a complex molecular code (Raush^ 
Barry, Hertel & Swain, 197A). 

Comparative Coding Procedures 

After training was completed, each coder was randomly 
assigned 

to cede the individual family problem solving situations. Both 
coders were on-site to code the problem solving interactions; 
one used the 083 instrument and the other the paper and pencj.1 
technique. The family interaction for each problem vignette 
was also videotaped. A different research assistant 
transcribed the two problem solving interactions during the 
next week following the on-site coding. Problem solving 
situation #1 involved 6 minutes AO seconds of interaction 
whereas problem solving interaction #2 constituted 5 minutes 20 
seconds. During the next two weeks, the remaining coding and 
recording methods were completed. Some bias is built into this 
method in that the coders were exposed to the same family 
interaction several times and thus may have improved in their 
level of reliability, however using additional coders would 
have required costly training. Thl ^lay have been minimized by 
the fact they were also coding other families during this same 
period . 
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Results 

Coding was completed as planned with one exception. During 
the on-site coding an editing problem developed with the use of 
the 0S3 which made it necessary to drop this technique from the 
on-site analysis. 

Kumber of Ratings 

All techniques using transcripts had a defined number of 
coding units, 113 in situation 1, and 108 in situation 2# 

Place Table 1 about here 

For the remaining recording methods, the fewest number of units 
were coded using the on-site paper and pencil recording method. 
The number of units coded in situation #1 was 37% of the 
identified units on the transcript. For situation #2, the 
percentage was A7Z. The recorders identified slightly core 
coding units using video than they did in direct observation. 
Using the paper and pencil recording method with video the 
recorders obtained 7831 and 87Z of the units identified in the 
transcript for situations 1 and 2. Using the 0S3 recording 
device with video the percentages were 70% and 66% 
respectively. 

Interrater Reliability 

Cohen*8 (1960) coefficient of agreement was calculated for 
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each recording method using a transcript:. Each coder's rating 
was compared to the ratings arrived at by consensus. Table 2 
reports the data. This method is a conservative estimate of 
reliability compared with simple percentage agreement. 
Reliabilities were very high and may reflect some inflation due 
to the repeated viewing of transcripts or videos. 

Place Table 2 about here 

Time Requirements 

Table 3 summarizes the time estimates arrived at for the 
differing methods. On-site coding was the most economical, 
time-wise. Use of video without transcript and a paper and 
pencil recording method involved a minimum of ten minutes to 
set up the video and at least three passes of the tape for 
coding. Using a mechanical recording device (0S3) had an 
additional time requirement to edit where necessary and to dump 
the data to a storage tape (30 minutes). 

Methods requiring a transcript required much more time. 
Transcribing is time consuming. We have estimated it takes a 
minimum of 2 1/2 hours to produce a clean, accurate copy of up 
to 10 minutes of tape. In addition it takes 30 minutes to 
unitize the transcript . An experienced coder can code up to 
115 items in approximately 30 minutes. Using a transcript as 
well as video adds the possibility of re-reviewing the tape 
while coding. We estimate this takes about 4 to 5 times the 
tir.e estimate for video alone. 
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Consensus coding demands that coder's continue discussing 
an Item until a similar coding category Is agreed upon. Based 
on a disagreement rate of approximately 15Z and discussion time 
of 3 minutes per coding unit, an additional one hour of coding 
time was added beyond that reported for transcript and video. 
For researchers, such time requires additional funding. 



Profile Ccmparlslons 

Data comprising the entire 26 category problem solving code 
were summarized into seven summary categories: S^i Fragments; 
S : Identification of Problem; Alternative Generation, 

S • Evaluation; Resolution Mechanisms; S : Proposed 

Decisions; S^: Meta-problem solving. Relative percentages 
based upon total units in the summary code compared with the 
total units coded were calculated for each summary code. This 
represented the family profile used to compare the recording 
methods (Tables 4 & 5) • 

The methods using transcripts revealed profile results 
which were remarkably similar: The video data had some 
noteable discrepancies when compared with the transcript codes 
particularly for summary codes S^, S^, Sj, and Sg but in 
general the differences were not great. The on-site coding was 
most discrepant with an overuse of certain codes (note the high 
percentage reported for compared with the other methods). 

Dlscusfilon 

The selection of a coding/recording method is a crucial 
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decision for a researcher studying family Interaction. Even an 
arm chslr comparison of methods reveals that less stringent 
recording methods have advantages. It would be expected that 
they would be less costly in terms of time and therefore money. 
Reduced time and money make the cost per subject less 
formidlble and allows a larger sampling of the population. 
This would be more appealing to funding agencies. The question 
which remains unanswered by such a cursary analysis is whether 
the researcher sacrifices quality data for the cost savings. 
The present paper provided an example of a systematic analysis 
which could be used to compare coding recording methods. The 
data profile of family problem solving interaction which 
resulted from ths use of different recording methods was 
compared in addition to several other variables. The profile 
results indicated that on-site observation and coding was the 
least desirable method. Not only were significantly fewer 
codes obtained but the data profile revealed was very different 
from that obtained by other methods. Training did not appear 
to ameliorate this discrepancy. Both of the on-site coders had 
been involved in an extensive training process which Involved 
training to identify the coding unit as well as to apply the 
code . 

This data contrasts with that reported by Bench> Hoffman 
and Wilson (197A) and Bench and Wilson (1975, 1976) comparing 
live versus video recording on neonates, 6 week-old and 6 
month-old infants responding to sound stimulation. They report 
that little or no information is lost using live observation 
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and In fact, obRervatlon of 6 month-olds may even be somewhat 
inferior using video. Sackett, Ruppenthal and Gluck (1978) 
reported that live recording catches 90% of Information 
extracted from repeated viewing of monkeys Interacting on a 
16mm film* 

Several factors could account for these differences. One 
is coder anxiety. On-site coding has an added element of coder 
anxiety in that this Is a once and for all chance to collect 
the data. In this study the coder had to make a decision 
regarding the unit to be coded as well as assigning one of 26 
possible problem solving codes to the unit. This involves 
split second decision making. The task may be more complex 
than in previously reported studies which observed more molor 
behavior. Another more Important factor is the fact that 3 
person family groups were being observed by one observer. When 
three persons are interacting interruptions and overlaps of 
communication naturally occur. The on-site recording and 
coding difficulties of poor sound transmission to the coder 
cannot be compensated for by a re-review. The factors may 
decrease both comprehensiveness (number of units coded) as well 
as accuracy (data profile)* 

Use of the video Increased the level of comprehensiveness 
as well as accuracy, even without a transcript. There were, 
however , some sacrifices in the use of the technique . It would 
appear that fragments (Incomplete statements ) were identified 
less frequently than in techniques using a transcript. There 
were also some small differences in reported proportions. 
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Ail of the methods using transcripts were most similar in 
comprehensiveness and accuracy. These were also the most 
cost ly • 

The systematic comparison of recording techniques reported 
in this paper provided support for the use of video without 
transcription for the present study of 3 person family problem 
solving interaction. Because the data were derived from a 
limited sample of approximately 100 behavioral units in each of 
two family problem solving interactions further comparisons are 
warranted before generalizing to other interactional research* 
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TABLE X 

Number of Ratings Compared Over Recordiag Methods 



Technique Number of Ratings 





Situation #1 


Situation ^2 


On-site 

0S3 (dropped) 




51 


Video 
0S3 

PflTiAT flni^ Penrll ( CoA&T 


79 
88 


71 
94 


Transcript Only 
Coder #1 
Coder #2 


113 
113 


108 
108 


Transcript and Video 

Paper and Pencil (Coder #1) 
Paper and Pencil (Coder #2) 


113 
113 


108 
108 


Transcript and Video 

Cons'&nsus - (Both Coders) 


113 


108 
* 



^Consensus coding is done jointly. 
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TABLE 2 

Interrater Reliability Estimates; Transcript Methods 







K 


K 


Coding Method 


Coder 


Situation #1 


Situation #2 


Transcript 


1 
2 


0.957 
.9148 


.905 
.911 


Video & Transcript 


1 
2 


.9680 
.8849 


.931 
.8074 
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TABLE 3 

Tlme^ Required by Coding Method 



Estimated Time 





Situation #1 


Situation #2 


On --Bite 
0S3 • 

Paper and Pencil 


6:40 


5:20 
5:20 


Video only* 
0S3 

Paper and Pencil 
*3 passes, 10 mln set up, dumping 
for 0S3 


50 : 00 
30:00 


46:00 
26:00 


Transcript only* 

Paper and Pencil Coder 1 
Paper and Pencil Coder 2 
*Includes transcribing, cleaning, 
unitizing, coding 


240:00 
210:00 


210:00 
180:00 


Transcript and Video* 

Paper and Pencil Coder 1 
Paper and Pencil Coder 2 


290:00 
270:00 


270:00 
260:00 


Transcript and Video* 

Consensus Code 
*Includes transcribing, cleaning, 

unitizing, coding, re-reviewing, 

and discussing 


360:00 


345:00 



Time is recorded in minutes and seconds 
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TABLE 4 

Relative Percentages of Codes by Summary Code 
by Different Recording Methods 

Situation #1 



Coding Method 


^1 


^2 


^3 






^6 




On Site 

0S3 (dropped) 
Paper and Pencil 


Coder 
#2 


0 


68 


0 


22 


10 


0 


0 


Video 
0S3 

Paper and Pencil 


#1 


13 
13 


38 
38 


11 
8 


23 
15 


10 
15 


5 
11 


0 
0 


Transcript Only 

Paper and Pencil 
Paper and Pencil 


#1 

#2 


22 
24 


38 
37 


4 
6 


17 
17 


9 
8 


10 
8 


0 
0 


Transcript and Video 
Paper and Pencil 
Paper and Pencil 


#1 
#2 


21 
22 


36 
38 


4 
6 


17 
15 


8 


10 


0 


Transcript and Video 
Consensus 


22 


40 


4 


16 


8 


10 


0 
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TABLE 5 

Relative Percentages of Codes by Suniuary Code 
by Different Recording Methods 

Situation #2 



Coding Method 


^1 


^2 


^3 


^4 


^5 


^6 


^7 


Coder 

Uu Ol 1. c 

0S3 

Paper and Pencil #2 


2 


67 


2 


22 


8 


0 


0 


Video 
0S3 

Paper and Pencil #1 


13 
11 


42 
39 


11 
5 


24 
20 


4 
11 


6 
14 


0 
0 


Transcript Only 

Paper and Pencil #1 
Paper and Pencil #2 


15 
13 


39 
44 


8 
5 


22 
22 


5 
5 


11 
11 


0 
0 


Transcript and Video 

Paper and Pencil #1 
Paper and Pencil #2 


11 
13 


43 
40 


3 
8 


22 
22 


7 
6 


14 
11 


0 
0 


Transcript and Video 
Consensus 


13 


41 


6 


23 


5 


13 


0 
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